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ABSTRACT 

Several predictors of occupational choice derived 
from a theory of careers (Holland, 1973) were G'^amined. Hypotheses 
that a person's competencies, activities, self-estimates, interests, 
and vocational choices can be organized by a six-category typology to 
understand and predict subsequent choice were tested. Samples cf 894 
men and 989 women took the Self-directed Search (SDS) at entry into 
two colleges and reported their occupational choices one or three 
years later. All hypotheses received support, although the 
predictions varied in efficiency. Other analyses indicated that 
current vocational choice is the best predictor vpf later choice, and 
that scores based on sex-specific norms are less efficient predictors 
than raw scores. The findings support both the theory and the 
validity of the SDS assessment for men and women. Extrapolations from 
these conclusions are limited by the nature of the criteria and the 
population tested (Collage students) . (Author/PC) 
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Introductory ^ta tement 



The Center for Social Organization of Schools has two primary 
objectives: to develop a scientific knowledge of how schools affect 
their students, and to use tliis knowledge to develop better school 
practices and organization. 

The Center works through three programs to achieve its objectives. 
The Schools and Maturity program is studying the effects of school, 
family, and peer group experiences on t)ie development of attitudes 
consistent with psychosocial maturity. The objectives are to formulate, 
assess, and research important educational 1 goals other than traditional 
academic achievement. The Schoo 1 Organi z ation program is currently 
concerned with authority-control structures, task structures, reward 
systems, and peer group processes in schools. llie Careers program 
(formerly Careers and Curricula) bases its work upon a theory of career 
development. It has developed a self-administered vocational guidance 
device and a se If -directed career program to promote vocational develop- 
ment and to foster satisfying curricular decisions for high school, 
college, and adult populations. 

This report, prepared by the Careers program, examines thi:ee 
issues in predicting vocational choices of men and women. 
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Vocational Choices of Men and Women: 
A Comparison of Predictors from the SDS 

This report examines three issues in the prediction of vocational 
choice. First, the comparative usefulness of self-report measures of 
interests, abilities, activities, and se If -e s tima te s for predicting 
voca tiona 1 choice is examined • Second , more evidence is provided that 
asking people about their occupational choices in an organized and 
simple manner is a useful way to get efficient predictive information. 
Third, the practical usefulness of sex-specific norms in assessing 
the vocational aspirations of men and women is considered. 

The first predictive issue entails a comparative analysis of the 
validity of components of the Se If -Pi re c ted Search (SDS) , a guidance 
simulation (Holland, 1972). Does each component of the SDS (activities, 
competencies, occupations, se 1 f- ra tings ) predict a student's subsequent 
vocational choice? Are the summary codes more efficient predictors 
than the individual components? Although there is substantial evidence 
that the components of the SDS have convergent and discriminant validity 
with respect to one another (Holland, 1968, 19 71'.), no single study has 
investigated these questions simultaneously and with the same criterion. 

The second predictive issue is the re la t ive usefulness of a person's 
current vocational choice for predicting subsequent choices. The belief 
that a person's stated choice is a use ful ^ face-valid predictor has never 
been widely held or expressed. This situation perhaps arose because of 
the way voca tiona 1 psycho logy deve loped . His torica 1 ly , psycholog 1 s ts 
were more concerned with establishing the validity of interest inven- 
tories, and the inve stigation of the validity of expressed choice 
received little attention. In addition, the potential Vc'ijlue of 



classification schemes for giving psychological meaning to choices and 

for making better predictions has only recently become apparent. In 

the last five years Holland and Lutz (1968) showed that when choices 

are grouped according to a classification scheme, prediction of later 

choices from earlier choices can be highly efficient. The Elt.on end Ros(j (1970) 

study and reviews by Dolliver (1969) and Whitney (1969) also support 

the usefulness of expressed choice in research and counseling. In 

spite of this positive evidence, counselors and investigators continue 

to make only modest use of this kind of information and instead favor 

more esoteric predictors or even advocate the use of two interest 

inventories rather than just one (Munley, Fretz & Mills, 1973). 

The third predictive issue stems from the belief that the use of 
interest and ability measures may be unfair to women because women 
have not had sufficient opportunity to develop interests and abilities 
in some areas. This assumption has led some authors to 

urge that special theories of careers be developed for women 
(Zytov7ski, 1969), or that norming of interest and ability scores by 
sex to statistically remove mean score differences may render the 
measures less biased (AMEG, 1973). Exploring the later suggestion, the 
SDS Summary Scores were normed so that the predictive validity of the 
high-point codes derived from standard scores and raw scores could be 
compa red . 

Method 

Sample 

The data were collected in an earlier study (Nafziger, Holland & 
Gottfredson, 1973) of student satisfaction in college. Volunteers came 
from two samples. One group of 2,508 students in freshman orientation 
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at a state university was assessed in the fall of 1970. A second 
group of I5I83 students undergoing freshman and transfer student 
orientation in a suburban public liberal arts college was assessed in 
the summer of 1972. The student body at the state university was more 
diverse than that at the liberal arts college where many students tradi- 
tionally prepare to be teachers. Questionnaires were sent to students 
of both groups who could be located in the spring of 1973 • After 
sending two reminders, a total of 7^6 questionnaires, including 62k 
with an expression of occupational choice , were obtained from the liberal 
arts group, and I5U87 questionnaires, including 1,259 with an expression 
of choice, were obtained from the state university. This was almost a 
three-year interval for the state university sample and about one year 
for the liberal arts sample. The reasons for nonresponse are not known, 
and representativeness is not claimed. In the state university sample, 
only the predictive validity of the SDS summary and normed summary 
scores could be assessed, because information about the SDS components 
was not available. 

Measures and analyses 

The source of all predictors was the SDS (Holland, 1972). 7nis 
device measures a person* s resemblance to six theoretical models by 
uapping vocational interests and their correlates. The types assessed 
by zhis inventory are named Realistic, Investigative, Artistic, Social, 
En*:erprisln^, and Conventional. These types are given their fullest 
description in Holland (1973)- The occupational daydreams portion of 
this simulation asks the respondent to report the occupations he has 
considered and to list the most recent one first. The first occupational 



daydream was coded according to its resemblance to the models by using a 
list of occupational titles (Holland, 1972). Itie activities portion 
asks the respondent to indicate those activities he or she likes to do. 
A typical item asks if the respondent likes to "nv3et important people," 
The competencies component asks what the person can do well or 
competently. Tne respondent is asked, for example, if he or she can 
describe the function of the white blood cells. The occupations part 
of the SDS asks which of a list of occupations interest or appeal to 
the respondent; sample iteuis are "composer" and "high school teacher." 
The two self-estimates components ask the respondent to rate himself on 
two sets of skills or abilities, for example, scientific ability and 
math ability. 

Each component yields a score for each of the six types. The type 
receiving the highest score for each component was the classification 
of that person for predictive purposes using that component. Each 
person was thus classified six times, once by each of the five components 
of the SDS usually scored, and once by the first occupational daydream. 
In addition, a summary score was obtained by using all components but 
the daydream section. For each component, the first-ranked score was 
weighted three, the second-ranked score was weighted two, and the 
third- ranked score was weighted one; these weights were then summed 
across all five components. The type receiving the highest score by this 
method was the sunimary code used as predictor. 

All ranking.^ of scores were performed by a computer program which 
listed tied scores in the order in which they appear in the assessment 
booklet. This order, which is presumably the way persons using the 
device typically break ties, was the order used for analyses when ties 
were not excluded. It is more conservative than the preferred methods 



of breaking ties in favor of the base rate or the respondent's judgment. 

Normed summary scores were calculated by subtracting the mean and 
dividing by the standard deviation for the appropriate sex using a large 
sample of college students. Means and standard deviations were obtained 
by pooling data from ten college samples collected earlier for various 
purposes. These samples are described elsewhere (Holland, Gottfredson & 
Nafziger, 1973; Nafziger, Holland & Gottfredson, 1973). The two samples 
studied in this report are included in the larger normative sample which 
is more diverse and representative. The standard scores were then 
ranked; the highest-ranked score defined the category used for prediction. 

On the follow-up questionnaire, respondents expressed their occupa- 
tional choices by selecting one of a coded list of 98 alternatives. 
Typical categories were "education administration/' "data processing," 
and "journalism." A total of 91 persons indicated that their field was 
not listed and 155 persons indicated that they were undecided; these 
were excluded from the analyses. 

Holland (1972) reports reliabilities of the scores on components 
of the SDS, but the reliabilities of classifications were required for 
this study. These reliabilities are not known, but the reliabilities 
of the SDS summary code and the daydreams have been approximated. Kappa 
(Cohen, I960) was calculated to assess reliability of the high-point 
code. Kappa is a coefficient of greater- than-chance agreement given 
the two marginal frequencies for a set of nominal groupings. Kappa may 
be interpreted as the proportion of above-chance possibilities for 
agreement actually occurring, and it is equal to 0 for dichotomous 
cases when the two marginal distributions are. the same. Its maximum 
possible value is 1.00. 
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Appropriate retest data were available for a sample of high school 
students tested two to three months after an initial testing. The 
value of kappa for 122 high school men was .62 and for 83 women was 
.42« Data to compute the reliability of the high-point code were also 
available for college students who had been asked the daydream question 
and who had earlier the same day responded to the following item: 
"List all the jobs or occupations you could do and would like, if you 
had enough money to get the necessary training, and if you could get 
that job when you finished your training or education." Between these 
two measures, kappa was .67 for 296 men and was .71 for 321 women. All 
these values are only rough estimates; the high school women's sample 
was especially small. No information is presently available on the 
reliabilities of types obtained in other ways, and the reliability of 
the occupational choice item is also not known. 

The analysis was performed by categorizing students according to 
each predictor (one of six types) and by their later choice (one of 
six occupational categories). Correct predictions occurred whenever 
the category of the occupation chosen had the same first letter as the 
predictor. Kappa was calculated for each table, and the probability 
of obtaining a value of kappa as large as that observed, given the 
assumption of no association, was calculated (Fleiss, Cohen &. Everitt, 
1969). 

Results 

Tables 1 and 2 show that about 407o of the men and 667o of the women 
in the liberal arts college expressed occupational choice in the 
category predicted by the SDS assessment one year earlier. Similarly, 
427o of the men and 587^ of the women in the state university chose 



occupations in the category predicted by the SDS assessment almost three 

Insert Tables 1 and 2 about here 



years earlier. These predictions were statistically significant and 
moderately efficient with all p's less than .001. Kappas range from 
.24 to .33. By predicting that all persons will later choose occupa- 
tions most often chosen by persons of their school and sex, 26.67o of 
the liberal arts men, 38.7% of the state university men, 72.47o of the 
liberal arts women, an- 61.87o of the state university women would have 
been correctly classified. Predictions for men, but not women, 
exceeded the base rate prediction. 

Table 3 shows that simply coding a person's most recent occupa- 
tional daydream using the six-category scheme was also significantly 
associated with expressed choice one year later for both men and 
women (p< .001). In addition, a person's occupational daydream was 
more efficient than was the SDS summary code for making predictions. 

Insert Table 3 about here 



Kappa equaled .41 and .44 for men and women respectively, and about 
127o more men and 3.57o more women fell into the diagonal colls when th^: 
daydreams rather than summary codes were used. 

Table 4 shows a comparison of all predictors derived from the SDS 
in its usual form, including self-reported competencies, ac t ivi t ie s , occupa tions 
and self-estimates. The general trend for all predictions was the same-- 
a disproportionate number of cases fell on the predicted diagonal. The 
first column of Table 4 shows the percent of choices falling into the 



predicted category for each predictor when ties were included. The 
second column shows that these values increased for all predictors when 
cases with tied high-point codes were excluded (daydreams are never 
tied). The third column shows Cohen's kappa based on data with ties 
included and is therefore conservative. All predictors were associated 
with expressed choice beyond chance levels but varied in efficiency. 



Insert Table 4 about here 



The occupational daydream was most efficient, followed by the occupations 
component for men and by the SDS suiranary for women. The competencies 
component appeared to be least efficient for men, and the second set of 
self-ratings was least efficient for women. 

When summary scores were normed separately by sex, the proportion 
of men choosing occupations in the predicted category stayed about the 
same--43.5% for the liberal arts group and 40.5% for the state univer- 
sity (kappa = .32 and .26, p*s<^.001). This was a slight decrease for 
men in the later group and a slight increase for men in the former. 
For women, however^ predictive validity decreased greatly. Only 28 . 77o 
wf the women in the liberal arts college and 32.8% of the women in the 
state university now chose occupations in the predicted categories. 
Nevertheless there was still an association between the standardized 
high-point code and the occupation chosen (kappa = .12 and .17, p's< .001). 

Insert Table 5 about here 



While a higher proportion of the few women choosing occupations classified 
as Realistic, Enterprising, and Conventional were correctly identified by 



the normed scores procedure, this small gain was overwhelmed by a massive 
decrease in the proportion of other types correctly identified. These 
results for women are shown in Table 5 and may be compared to the results 
in Table 2 for the unnormed scores. 

Discussion 

This study has two important limitations. The student samples do 
not represent any well-defined population, and the criterion (student's 
expressed occupational choice after one or three years) is an inter- 
mediate one. Actual job selection would be more persuasive. Despite 
these shortcomings, the levels of predictive efficiency found' are con- 
sistent with similar studies using more diverse or national samples 
(Holland & Lutz, 1968; Hughes, 1971; Nafziger, Holland, Helms & 
McPartland, 1972). In addition, the vocational choices of college 
students have a strong relation to the job followed after graduation 
(D>'er, 1932; Strong, 1953; Sharp & Krasnegor, 1966). 

The associations found be tween predications made from the separate 
components of the SDS--the activities, competencies, occupations, and 
self-estimates-'-and subsequent choice supports the theoretical notion 
that each class of personal attributes contributes to a person*s 
decision. This interpretation does not specifically support the 
developmental chain hypothesized by Holland (1973), but it is consistent 
with that view. 

In addition, the validity of all predictors supports their practical 
use ia the self-assessment of individuals desiring career guidance. This 
is particularly true in the case of the first set of self -estimates for 
women, which was a better predictor of subsequent choice than was the 
occupations component, a more direct measure of interests. In line with 



this finding Is the related observation that the SDS suironary code--a 
composite rating of interests, abilities, and activities--also predicted 
the criterion for women more efficiently than did the occupations 
component alone. 

The finding that the most recent occupational daydream was the 
best predictor of later choice from among all the predictors replicates 
and extends an earlier finding (Holland & Lutz, 1968) and is consistent 
with critical reviews of the literature (Dolliver, 1969; Whitney, 1969). 
There seems to be litule doubt that what a person says he wants to do 
is an important piece of information. The increased use of this infor7.ia- 
tion in counseling practice and research urged earlier (Holland & Lutz, 
1968; Whitney, 1969) is desirable. 

An examination of the base rate problem, elaborated by Meehl and 
Rosen (1955), is useful in understanding some of the results. Limita- 
tions of sampling and the predominance of women choosing Social occupa- 
tions in almost any sample make prediction for women difficult. About 
727o of the women in the liberal arts college and 62% of the women in 
the state university chose occupations classified as Social; that is, 
their choices were typified by helping or teaching occupations. This 
means that we would have been correct in ITU and 627o of the cases, 
respectively, if we had simply predicted that all women would choose 
those occupations. In fact we would have been right more often than 
by using any of the assessments used in this study. 

Prediction can be perfect only when the marginal distribution 
of predictions is identical to the marginal distribution of outcomes, 
although because of error or invalidity it generally will not be 
perfect even then. When the marginal distributions are very unequal, 
predictive validity suffers. In the samples of women, the c-ffcct of 
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employing sex-specific norms in order to make the SDS hij;h-point codes 
'^fairer" resulted in more rectangular distributions of predictors while 
the distribution of choices remained unchanged. As a result less 
efficient rather than more efficient predictions occurred (see Table 5). 

The effect of dissimilar marginal distributions can be demonstra- 
ted in another way. Using Cohen's (1960) formula, the maximum possible 
value of kappa, given the frequencies shown for the state university 
sample of women, is ,47 and is only .34 for the liberal arts college 
group of women. It may be noted that even the usual unnormed scoring 
procedure suffered slightly from unequal marginal distributions, with 
too few women classified in the Social category on the predictor. 

The evaluation of potential sex bias if a complex and elusive 
process. If one were to examine a matrix of correlations among the 
sets of scales in the SDS, such as the one in Holland (1972), one 
might conclude, as does Holland, that the pattern is evidence of the 
convergent and discriminant validity of these scales for measuring 
attributes of persons relevant to their careers. Alternately one 
might cite mean differences between the sexes on some of the scales 
and conclude that the scales measure cultural biases against men and 
women. When one also examines the predictive validity of the SDS com- 
ponents, however, a reasonable hypothesis is that the cultural bias whi.ch 
does exist is validly measured, A corollary of this hypothesis is that 
if the bias ceased to exist so would the mean differences between the 
sexes on these measures. This hypothesis constitutes a synthesis of 
the two seemingly contradictory views of the data. The attempt to use 
norms to adjust the scores of women on the basis of their sex succeeds 
in '^decreasing bias" to the extent that the measures do not do what 
they were intended to do--measure differences of individuals related 
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to occupational choices. 

Numbers alone may not fully convey the meaning of these results. 
To make the point in another way, consider the case of one of the 
volunteers in this study. Ms. S was 18 when she took the SDS. When 
she entered the liberal arts college, she planned to major in elementary 
education. Elementary teaching was probably not a big step up the 
social ladder for Ms. S because both her parents had college degrees. 
Elementary education is coded SAI (Social, Artistic, and Investigative 
in decreasing order of salience) in the Holland system. S's occupational 
daydreams were in line with her major at entry to college. In reverse 
order they were coded SAI, AIR, CEI, SAI, and SAE. When asked the next 
year, Ms. S was still majoring in elementary education with an A average. 
By use of ■ i;ex-3pec ific norms, Ms. S's Realistic score was increased 
because she was a woman, and her norm-based code became RAE--very 
different from the SAI code she earned when the usual scoring procedure 
was used. Hie occupations closest to the RAE code in the Holland system 
are compositor, typesetter, and bookbinder, all of which require manual 
and mechanical skills and virtually no social skills. In view of what 
we know of Ms. S, these are very inappropriate occupations for her. 

The choice of scoring methods to be used would appear to depend in 
part on whether one desires to work for the person seeking guidance or 
to work for social change. 

To reiterate, the four most important outcomes were: (a) the best 
pred iction of the group of occupations a person chose was made by simply 
asking and then classifying the response accor<ling to an or^^anizing 
scheme; (b) the hypotheses derived from Holland's theory of careers that 
a person's reported interests, competenclas , preferred activities and 
self-ratings are valid predictors of later expressed choice were generally 
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supported, although these predictors varied in efficiency; (c) the 
problem of validity in predicting choice for women conflicts with the 
good-mtentioned wish to avoid stereotyped classification because of 
a massive base rate problem; and (d) the attempt to use sex-specific 
norms for women was not successful. Extrapolations from these 
conclusions are limited by the nature of the criterion and the college 
sample s • 
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Table 1 



Validity of SDS Sunn-nary Code for Predicting 
Men's Expresoed Occupational Choices One and Three Years Later 



First Letter of 
SDS Summary Code 



Men in a State Liberal Arts College (N=192) 
Occupational Choice One Year Later 



R 


I 


A 


S 


E 


C 


Total 


4 


4 


• 4 


2 


8 


6 


28 


1 


21 


4 


12 


3 


6 


47 




4 


16 


7 




1 


28 


1 


10 


9 


27 


15 


3 


65 




2 


5 


3 


6 


1 


17 




3 


2 






2 


7 


6 


44 


40 


51 


32 


19 





R 
I 
A 
S 
E 
C 

Total 

39.6?o correct prediction, kappa = .24, p<.001; 26.6?„ correct 
prediction when all are classified S. 





Men 


in a 


State 


University 


(N=702) 




First Letter of 


Occupational Choice 


Three 


Years Later 


SDS SunHnary Code 




I 


A 


S 


E 


C 


Total 


R 


35 


43 


10 


13 


.24 


13 


138 


I 


25 


181 


20 


19 


27 


21 


293 


A 


2 


6 


17 


12 


11 


1 


49 


S 


8 


36 


10 


22 


47 


16 


139 


E 


2 


6 


9 


7 


34 


12 


70 


C 


1 




1 




5 


6 


13 


Total 


73 


272 


67 


73 


148 


69 





42.0?o correct prediction, kappa = .24, p< .001; 38.77o correct 
prediction when all are classified I, 



Note. --In this and succeedirig tables, the italicized entries are 
frequencies in the predicted cells. 



Table 2 



Validity of SDS Summary Code for Predicting 
Women's Expressed Occupational Choices One and Three Years Later 



First Letter of 
SDS Surmnary Code 



Women in a State Liberal Arts College (N=432) 
Occupational Choice One Year Later 

R I A S E C Total 



R 




1 




3 




1 


5 


I 




22 


1 


34 


2 


2 


61 


A 




10 


30 


33 


1 


2 


76 


S 


1 


21 


12 


234 


6 


3 


277 


E 






1 


3 






4 


C 








6 


2 


1, 


9 


Total 


1 


54 


44 


313 


11 


9 




66.4% 


correct 


prediction , 


kappa = 


.33, 


p<: • 


001; 72 



prediction when all are classified S» 



First Letter of 
SDS Summary Code 


Women in a State University (N=557) 
Occupational Choices Three Years Later 


R I 


A 


S 


E 


C 


Total 


R 


1 




1 






2 


I 


3 41 


6' 


34 


6 


2 


92 


A 


11 


27 


48 


10 


1 


97 


S 


29 


34 


247 


18 


5 


333 


E 


3 


4 


6 


2 


1 


16 


C 




2 


8 


1 


6 


17 


Total 


3 85 


73 


344 


37 


15 





58.0% correct prediction, kappa ^ .28, p < .001; 61,8% correct 
prediction when all are classified S. 



Table 3 



Validity of Occupational Daydream for Predicting 
Expressed Occupational Choices One Year Later 



Men in a State Liberal Arts College (N=163) 
First Letter of Occupational Choice One Year Later 



of Daydream Code R I A S E C Total 



R 5 2 4 4 3 4 22 

I 1254523 40 

A 2 18 11 3 34 

S . 5 2 23 5 35 

E 3 4 1 9 6 23 

C 1 12 5 9 

Total 6 38 32 45 24 18 

52.17<, correct prediction, kappa = .41, p < .001 



Women in a State Liberal Arts College (N=396) 
First Letter Occupational Choices One Year Later 

of Daydream Code R I A S E C Total 

R ~ 2 i 3 

I 34 4 39 1 4 82 

A 1 5 29 28 2 65 

S 7 3 21? 4 1 227 

E 5 4 1 2 12 

C 2 2 2 1 7 

Total 1 48 41 287 10 9 

69.97o correct prediction, kappa = .44 p < .001 



Table 4 

Summary of Validities of SDS Components for Predicting 
Expressed Occupational Choice One Year Later 



Men (N=192) Women (N=432) 

7c % hits % 1 hits 

Predictor hits no tie kappa hits no tie kappa 

Most frequent 

category 26.6 72.4 

First 

Daydream 52.1 .41** 69.9 .44** 

Activities 

Code 32.3 38.6 .17** 52.8 57.5 .17** 

Competencies 

Code 25.5 27.3 .06* 55.3 61.5 .13** 

Occupations 

Code 43.2 44.4 .30** 51.6 55.6 .21** 

First Set 

Self Ratings 38.5 43.6 .25^'^> 59.2 62.6 .36** 
Second Set 

Self Ratings 28.6 32.4 .12** 49.5 55.0 .14** 
Summa ry 

Code 39.6 44.0 .24** 66.4 69.4 .33** 



Note.--N=163 for men and 396 for women for the occupational daydream 
item because not all respondents completed that item. N*s fall as 
low as 145 for men and 353 for women when ties are excluded. The 
value of kappa is based on tables in which tied predictors were 
included . 

* p<.05 **p<.001 



Table 5 



Validity of Highest Normed Summary Score for Predicting 
Women's Occupational Choice One and Three Years Later 



Highest 




Women in 


a State 


Libe 


ral Arts 


College 


(N=432) 


Normed 




Occupational Choice One Year 


Later 




Score 




R I 


A 


S 


E 


C 


Total 




R 


1 7 


6 


34 


1 


2 


51 




I 


21 


1 


31 


1 


2 


76 




A 


11 


24 


54 


1 


2 


92 




S 


5 


5 


74 






84 




E 


2 


7 


46 


1 




56 




C 


8 


1 


54 


7 


3 


73 




Total 


1 54 


44 


313 


11 


9 





28.7% hits, kappa = .12, pC.OOl 



Highest 




Women in 


a State 


University 


(N=557) 




Normed 




Occupational Choice Three Years Later 


Score 




R I 


A 


S 


E 


C 


Total 




R 


2 9 


9 


27 


3 




50 




I 


1 51 


4 


49 


5 


2 


112 




A 


8 


30 


62 


10 


1 


111 




S 


6 


8 


81 


7 


1 


103 




E 


10 


18 


78 


9 


1 


116 




C 


1 


4 


47 


3 


10 


65 




Total 


3 85 


73 


344 


37 


15 





32.8% hits, kappa = .17, p^.OOl 



